Method and system for three-dimensional surface measurement with a mobile device

ABSTRACT

A three-dimensional (3D) imaging system includes a mobile device that has a display screen configured to display a series of patterns onto an object that is to be imaged. The mobile device also includes a front-facing camera configured to capture reflections of the series of patterns off of the object. The system also includes a controller that is configured to control a timing of the series of patterns that appear on the display screen and activation of the front-facing camera in relation to the appearance of the series of patterns.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the priority benefit of U.S. ProvisionalPatent App. No. 62/877,401 filed on Jul. 23, 2019 and U.S. ProvisionalPatent App. No. 63/007,524 filed on Apr. 9, 2020, the entire disclosuresof which are incorporated by reference herein.

REFERENCE TO GOVERNMENT RIGHTS

This invention was made with government support under Grant No.PR-258900-18 awarded by the National Endowment for the Humanities. Thegovernment has certain rights in the invention.

BACKGROUND

Three-dimensional (3D) imaging techniques are being used in a multitudeof scientific and commercial disciplines. Industrial 3D inspection,factory automation, agricultural imaging, medical 3D imaging, and 3Ddocumentation and analysis of art or cultural heritage artifacts areonly a few examples of the broad range of applications. The greatpopularity of 3D imaging is based on several advantages as compared totraditional two-dimensional (2D) imaging. Compared to a simple 2D image,a three-dimensional object representation is invariant to objecttranslation and rotation, as well as variations in surface texture orexternal illumination conditions.

SUMMARY

An illustrative three-dimensional (3D) imaging system includes a mobiledevice that has a display screen configured to display a series ofpatterns onto an object that is to be imaged. The mobile device alsoincludes a front-facing camera configured to capture reflections of theseries of patterns off of the object. The system also includes acontroller that is configured to control a timing of the series ofpatterns that appear on the display screen and activation of thefront-facing camera in relation to the appearance of the series ofpatterns.

An illustrative method for performing three-dimensional (3D) imagingincludes displaying, on a display screen of a mobile device, a series ofpatterns onto an object that is to be imaged. The method also includescapturing, by a front-facing camera of the mobile device, reflections ofthe series of patterns off of the object. The method further includescontrolling, by a controller that is in communication with the mobiledevice, a timing of the series of patterns that appear on the displayscreen and activation of the front-facing camera in relation to theappearance of the series of patterns.

Another illustrative three-dimensional (3D) imaging system includes afirst mobile device that includes a display screen that is configured todisplay a series of patterns onto an object that is to be imaged. Thesystem also includes a second mobile device that has a rear-facingcamera and a controller. The rear-facing camera is configured to capturereflections of the series of patterns off of the object. The controllerthat is configured to control a timing of the series of patterns thatappear on the display screen and activation of the rear-facing camera inrelation to the appearance of the series of patterns.

Other principal features and advantages of the invention will becomeapparent to those skilled in the art upon review of the followingdrawings, the detailed description, and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments of the invention will hereafter be describedwith reference to the accompanying drawings, wherein like numeralsdenote like elements.

FIG. 1 depicts a Phase-Measuring Deflectometry (PMD) system inaccordance with an illustrative embodiment.

FIG. 2 is a block diagram depicting a 3D imaging system in accordancewith an illustrative embodiment.

FIG. 3A is a block diagram depicting components of the mobile device ofthe 3D imaging system in accordance with an illustrative embodiment.

FIG. 3B depicts handheld use of the mobile device to measure a portionof stained glass surface in accordance with an illustrative embodiment.

FIG. 4A depicts horizontal fringes within the effective measurementfield for a planar object (i.e., plexiglass plate) in accordance with anillustrative embodiment.

FIG. 4B depicts a horizontal fringe pattern reflected from the surfaceof plexiglass plate in accordance with an illustrative embodiment.

FIG. 4C depicts a calculated phase map based on the reflected horizontalfringe pattern in accordance with an illustrative embodiment.

FIG. 4D depicts a calculated gradient map after high pass filtering inaccordance with an illustrative embodiment.

FIG. 5A depicts a first stained glass test tile in accordance with anillustrative embodiment.

FIG. 5B depicts a second stained glass test tile in accordance with anillustrative embodiment.

FIG. 5C depicts a third stained glass test tile in accordance with anillustrative embodiment.

FIG. 5D depicts a fourth stained glass test tile in accordance with anillustrative embodiment.

FIG. 6A depicts a surface normal map calculated from measurements of thefirst stained glass test tile in accordance with an illustrativeembodiment.

FIG. 6B depicts a surface normal map calculated from measurements of thesecond stained glass test tile in accordance with an illustrativeembodiment.

FIG. 6C depicts a surface normal map calculated from measurements of thethird stained glass test tile in accordance with an illustrativeembodiment.

FIG. 6D depicts a surface normal map calculated from measurements of thefourth stained glass test tile in accordance with an illustrativeembodiment.

FIG. 7A depicts normal map measurement results with room light for thesecond and third stained glass test tiles in accordance with anillustrative embodiment.

FIG. 7B depicts normal map measurement results for the second and thirdstained glass test tiles from handheld measurements in accordance withan illustrative embodiment.

FIG. 8 depicts a circular shaped stained glass structure imaged usingregistration in accordance with an illustrative embodiment.

FIG. 9A shows a captured white image from a first view in accordancewith an illustrative embodiment.

FIG. 9B shows a captured white image from a second view in accordancewith an illustrative embodiment.

FIG. 9C depicts detected and mapped features in the two white images inaccordance with an illustrative embodiment.

FIG. 9D depicts the registered white images in accordance with anillustrative embodiment.

FIG. 10A depicts a normal map of the stained glass surface acquired froma first viewpoint in accordance with an illustrative embodiment.

FIG. 10B depicts a normal map of the stained glass surface acquired froma second viewpoint in accordance with an illustrative embodiment.

FIG. 10C depicts a normal map of the stained glass surface under a firstalternative shading in accordance with an illustrative embodiment.

FIG. 10D depicts a normal map of the stained glass surface under asecond alternative shading in accordance with an illustrativeembodiment.

FIG. 11A is an image of a painting being imaged in accordance with anillustrative embodiment.

FIG. 11B depicts a first view of the surface shape of the marked regionin FIG. 11A in accordance with an illustrative embodiment.

FIG. 11C depicts a second view of the surface shape of the marked regionin FIG. 11A in accordance with an illustrative embodiment.

FIG. 12A depicts an image of a key in accordance with an illustrativeembodiment.

FIG. 12B depicts a measured normal map of the key of FIG. 12A inaccordance with an illustrative embodiment.

FIG. 12C depicts normal maps of a nickel and a dime in accordance withan illustrative embodiment.

FIG. 12D depicts an image of a circuit board in accordance with anillustrative embodiment.

FIG. 12E depicts a measured normal map of the circuit board of FIG. 12Din accordance with an illustrative embodiment.

FIG. 13A depicts a pattern of water drops on an enameled surface inaccordance with an illustrative embodiment.

FIG. 13B depicts a normal map of the pattern of water drops depicted inFIG. 13A in accordance with an illustrative embodiment.

DETAILED DESCRIPTION

In traditional systems, the benefits of 3D imaging, as compared to 2Dimaging, do not come without a price. Three-dimensional imageacquisition is not as easy or as straightforward as taking a 2D snapshotwith a mobile phone camera. As a first consideration, one must pick theappropriate 3D imaging technique for the object to be measured. Thisdecision is strongly dependent on the microscopic surface structure ofthe object, which can be roughly divided into two categories: diffuse(or scattering) and specular. Diffusely scattering surfaces are commonlymeasured by projecting a temporally or spatially structured light beamonto the object and evaluating the back-scattered signal. Time-of-Flight(ToF) imaging and Active Triangulation (or Structured Light) imaging areprominent examples. Another imaging procedure is referred to asPhotometric Stereo, where the object surface is sequentially floodilluminated with point light sources from different angles.

Unfortunately, the application of the aforementioned imaging principlesto specular surfaces yields only limited success. The reason for this isstraightforward in that specular reflections from a point light sourcescarcely find their way back into the camera objective. Depending on thedistribution of surface normals with respect to light source and cameraposition, only a few sparse (and probably overexposed) specular spotsmay be visible in the camera image. The same problem is also common forinterferometric instruments, which can accurately measure smoothlyvarying specular surfaces with high precision, but fail for specularsurfaces with a large angular distribution of surface normals.

One solution to this problem is to extend the angular support of theillumination sources. This is the basic principle behind deflectometry,where a patterned screen replaces point-like light sources, as shown inFIG. 1. Specifically, FIG. 1 depicts a Phase-Measuring Deflectometry(PMD) system 100 in accordance with an illustrative embodiment. Asshown, a screen 105 with a fringe pattern is positioned over thespecular (or reflective) surface of an object 110 that is being imaged.A camera 115 is used to capture reflections off of the object 110. Thesurface normal of the object 110 is indicated by an arrow. A normal mapof the surface of the object 110 can be calculated based on thedeformation of the fringe pattern in the captured camera image.

In a PMD system, the screen can be self-illuminated (e.g., a televisionmonitor, computer screen, etc.) or printed. In deflectometry systems,the screen and camera both face the object, which means that the cameraobserves the specular reflection of the screen over the surface of theobject being imaged. The observed pattern in the camera image is adeformed version of the image on the screen, where the deformationdepends on the surface normal distribution of the object surface, asshown in FIG. 1. From this deformation, the normal vectors of thesurface can be calculated. To calculate a normal vector at each camerapixel, correspondence between camera pixels and projector pixels isdetermined. A common technique to achieve this is with thephase-shifting of sinusoidal fringes. The resulting Phase-MeasuringDeflectometry (PMD) has established itself as powerful technique that isused with great success in industrial applications, e.g. to test thequality of optical components or to detect defects on metallic partslike car bodies. Given a proper calibration, it has been shown that PMDreaches precisions close to interferometric methods.

The PMD imaging procedure is just one of a number of techniques thathave been introduced to measure the 3D surface of specular objects. Mostof these techniques are closely related to PMD, but differ in themechanism used to establish correspondence between the screen andcamera. In principle any known pattern can be used in place ofsinusoidal fringes. Furthermore, the pattern does not even have to beself-illuminated, and will be effective in estimating surface shape aslong as there is some prior knowledge of the pattern. For example, somesystems utilize the reflection of color coded circles observed bymultiple cameras (which also resolves the bas-relief ambiguity). Othersystems use self-illuminated screens with patterns such as stripes,multiple lines, or even a light field created from two stacked LEDscreens. Additionally, screenless methods can be used to analyzeenvironment illumination or track prominent features (e.g. straightlines) in the environment to obtain information about the slope ofspecular surfaces.

Each of the above-mentioned techniques comes with benefits anddrawbacks. For example, some of the techniques that use a static patterninstead of a phase shifted sinusoid are capable of single-shotacquisition. However, this comes at the cost of lateral resolutionand/or restricts the surface frequencies that can be measured.Self-calibrating photometric stereo techniques use known reflectancemaps of object surfaces to measure their 3D structure. Such approachescan be used for partially specular surfaces, but fail when the surfaceis too shiny. Other techniques exploit sparse specular reflectionsproduced by photometric stereo measurements for 3D surfacereconstruction or refinement.

However, traditional PMD and other imaging systems are limited in theirability to perform 3D imaging of various surfaces. As one example,currently available techniques are unable to perform optimal 3Dmeasurement and analysis of stained glass surfaces, which can be foundin larger glass artworks, church windows, or glass reliefs. Such glasssurfaces can be composed of hundreds or thousands small colorful glasspieces that are connected with a metal frame. While generally flat innature, the shape of these small glass pieces is typically not entirelyflat. Over the centuries, several glass manufacturers have developed amultitude of techniques to imprint unique three-dimensional structuresto the glass surface that reflect and diffract light in a very distinctway. These unique 3D structures can be exploited to match the smallglass pieces in a stained glass painting to the individual manufacturersand to trace the circulation of stained glass and the respectivehistorical influence of the manufacturer around the globe. The latter isof significant interest for the cultural heritage community.

The task of digitizing 3D surface types similar to the surfaces ofstained glass leads to several fundamental and technical challenges forthe 3D metrologist. First, the objects (e.g. church windows) are largeand usually not portable. This makes it nearly impossible to transportthem to a controlled lab environment for surface measurement. Second,the surfaces to be measured often contain high spatial frequencies andthus a large variation of surface normals. This requires a high spatialresolution and angular coverage of the 3D surface measurement device.Third, the backsides of the objects are largely inaccessible, whichrules out all spectroscopy-like methods to characterize the 3D surface.Moreover, some objects can be highly opaque.

The inventors have determined that all of the above challenges can beaddressed by using a novel PMD measurement technique. However, most PMDsetups are bulky and cannot be applied in the wild. This bulkiness ismainly caused by the large screens used, which are intentionally chosento provide a large angular coverage and enable measurement over a largerange of surface normals.

A proposed solution to this problem is to use mobile devices (e.g., asmart phone, tablet, portable music player, etc.) for PMD measurements.Specifically, the screen of the mobile device can be used to display thepatterns and the front-facing camera of the device can be used to imagethe object surface. Since the screen size of mobile devices is limited,only a small angular range of surface normals can be measured in anysingle view. To overcome this limitation, the proposed systems utilizefeature-based registration, applied to multiple views acquired fromdifferent viewing angles. The features are extracted directly from theglass (or other) surface being imaged so that external markers orfiducials are not necessary.

Described herein are systems and methods for the three-dimensionalmeasurement of extended matte surfaces and extended specular surfaceswith high surface normal variations. In some embodiments, the proposedsystem utilizes a mobile hand held device and exploits the screen andfront camera of the mobile device for deflectometry-based surfacemeasurements. For reflective surfaces, a specific pattern (e.g.sinusoidal fringes) is displayed on a screen of the mobile device andthe surface slope is evaluated via deflectometry. For matte surfaces, apattern with low frequency (e.g. a gray-wedge) is displayed on thescreen of the mobile device and the surface slope is evaluated viamethods that are related to photometric stereo. In one implementationfor matte surfaces, an intensity gradient technique can be used in whichthe screen intensity is linearly ramped up through a gradient range(e.g., 0-255).

Using the proposed system, high quality measurements have beendemonstrated without the need for an offline calibration procedure. Inaddition, described herein is a multi-view technique to compensate forthe small screen of a mobile device such that large surfaces can bedensely reconstructed in their entirety. The proposed system utilizes aself-calibrating deflectometry procedure capable of taking 3D surfacemeasurements of specular objects in the wild (i.e., outside of alaboratory setting).

The proposed system is also accessible to users with little to notechnical imaging experience. For example, the proposed system canprovide 3D surface measurement capability to museum conservators,tourists, and other end users who may require an imaging system thatoffers extreme ease of use and minimal computational requirements. Tomake the system widely accessible, the inventors have developed aplatform that allows for a server-side evaluation of the captured 3Ddata in one embodiment. In such an implementation, the mobile device isused only to capture images and to display the evaluated data.

Described below are the proposed image acquisition and processingoperations that enable uncalibrated 3D Deflectometry measurements withmobile devices. Also described is a set of qualitative surfacemeasurements that are sufficient to identify and compare characteristicsurface structures, e.g. in stained glass surfaces. Additionallydescribed herein are the extension of the proposed methods and systemsto other (partially) specular surface types, such as oil paintings,technical parts, etc.

FIG. 2 is a block diagram depicting a 3D imaging system 200 inaccordance with an illustrative embodiment. The 3D imaging system 200includes a mobile device (or measurement device) 205, a controllerdevice 210, and a server 215. In alternative embodiments, the 3D imagingsystem 200 can include fewer, additional, and/or different components.For example, the 3D imaging system 200 can also include a portablemounting system that is configured to hold the mobile device in aspecific position and orientation relative to an object being imaged.The mobile device 205 can be a cellular phone, a tablet, a laptopcomputer, a global positioning system (GPS) device, a portable gamingdevice, a music player, or any other type of mobile computing device. Asdiscussed in more detail below, the mobile device 205 is used as themeasurement device that displays a pattern on its screen. The pattern isdirected toward an object being imaged, and a front-facing camera of themobile device 205 is used to capture data/images which can be used toform a 3D image of the object. While many of the embodiments describedherein include a mobile device, it is to be understood that the imagingtechniques described herein can be used with any computing device thatincludes a screen and a camera. For example, the proposed imagingtechniques can be implemented on stationary devices such as a desktopcomputer, kiosk computer, etc. that includes a screen and a camera.Thus, in some embodiments, the mobile device 205 can be replaced by astationary computing device.

The controller device 210 can be a mobile device similar (or identical)to the mobile device 205. Alternatively, the controller device 210 canbe a stationary device such as desktop computer. In one embodiment, thecontroller device 210 can be a small remote control (similar to atelevision remote) that is used to control operation of the mobiledevice 205. The controller device 210 can be local to or remote from themobile device 205, depending on the system. The controller device 210 isused to control the screen and at least a front facing camera of themobile device 205 such that the mobile device captures image data and/orimages of a surface of an object. Specifically, the controller device210 starts and monitors the measurement conducted by the mobile device205. The controller device 210 can also be used to display a finalresult of the measurement. The server 215 is used to process the imagedata and/or images of the surface of the object captured by the mobiledevice 205 such that a 3D image can be formed. Besides being able toprovide higher performance as compared to a mobile device, evaluatingthe data on the server 215 can provide other important benefits. Forexample, code changes (updates) can be directly made on the server 215without the need for a user to install a new version of the measurementapplication on the mobile device 205. Moreover, the server 215 can storethe evaluated data and work as a database, e.g. for the identificationof stained glass pieces or similar fingerprint applications in whichartwork, artifacts, etc. are being identified.

In an illustrative embodiment, the mobile device 205, the controllerdevice 210, and the server 215 can communicate with one another via anetwork 220, which can be the Internet, a cellular network, a local areanetwork (LAN), a wide area network (WAN), etc. Additionally, any of themobile device 205, the controller device 210, and the server 215 cancommunicate directly with one another through a wired or wirelessconnection, such as a Bluetooth® connection, etc.

FIG. 3A is a block diagram depicting components of the mobile device 205in accordance with an illustrative embodiment. As shown, the mobiledevice 205 includes a processor 300, a memory 305, a display screen 310,a transceiver 315, one or more cameras 320, and an application 325. Themobile device 205 can include fewer, additional, and/or differentcomponents in alternative embodiments. The controller device 210 and theserver 215 can similarly include at least a processor, a memory, atransceiver, application(s), a display screen, etc. The processor 300can be any type of computer processing component known in the art suchas a single core processor, a multiple core processor, a microprocessor,etc. The memory 305, which can be any type of computer storage, is usedto store data and algorithms which are used to perform computingfunctions. The memory 305 can also be used to store the application 325.The display screen 310 can be any type of display known in the art, suchas a light-emitting diode (LED) screen, liquid crystal display (LCD),etc. The transceiver 315 is used to facilitate communication directlybetween devices and/or through the network 220. The transceiver caninclude elements to facilitate communication via wired connection,wi-fi, Bluetooth®, etc. The one or more cameras 320 include at least afront-facing camera that is used to capture images of and/or dataregarding a surface being analyzed.

The application 325 is a dedicated program that enables the system togenerate 3D images. The application 325, or forms thereof, can beincluded on each of the mobile device 205, the controller device 210,and the server 215. The application 325 handles the image acquisitionprocess and manages data transfer between the server 215, the mobiledevice 205, and the controller device 210. FIG. 3B depicts handheld useof the mobile device 205 to measure a portion of stained glass surfacein accordance with an illustrative embodiment. The reflections of thescreen of the mobile device 205 are visible on parts of the glasssurface and are used to reveal its 3D structure. The measurement result,which is a normal map of the boxed area, is displayed in the zoomedinset of FIG. 3B. This procedure is described in more detail below.

In one embodiment, during image acquisition, the controller device 210causes the mobile device 205 to display phase-shifted sinusoidalpatterns and observe an object with its front-facing camera. The mobiledevice 205 can be positioned approximately 200 millimeters (mm)above/over the surface of the object. Alternatively, different distancescan be used based on the screen and optical properties of the camera ofthe mobile device 205. Because PMD is a multi-shot technique, a sequenceof temporally acquired images is used to calculate one 3D image. Duringthe measurement, the display of the mobile device 205 can project fouror more 90°-phase-shifted versions of a sinusoid in the horizontal andvertical directions, respectively. There is no limit to the number ofphase-shifted versions that can be used. Alternatively, less than 4(e.g., 3) phase-shifted versions of the sinusoid may be used. Differentfrequencies of the sinusoid can optionally be used instead of phaseshifted signals. In another alternative embodiment, instead ofsinusoids, a different pattern may be used such as a pattern of dots, acheckerboard pattern, a pattern of lines, etc. The position of themobile device relative to the object should remain fixed during thewhole acquisition process. Depending on the speed of projection andimage acquisition, this can be a hard task for an inexperienced user, ifa handheld measurement is desired. For an optimal measurement result,the mobile device can be fixed relative to the object being imaged witha mount. Alternatively, a free-hand guided single-shot principle can beused, as discussed below.

In an alternative embodiment, the 3D imaging system may not include aseparate controller device. In such an implementation, the controls canbe built into the mobile device 205 itself. In one implementation, aportion of the screen of the mobile device can be dedicated to controlsfor the system. These controls can enable displaying various patterns onthe remainder of the screen and capturing images with the front-facingcamera of the reflections of those patterns off of the object beingimaged. However, in order to maximize the area that can be imaged in asingle shot, it is desirable to use the entire screen of the mobiledevice to project the patterns (e.g., phase shifted sinusoidal patterns)onto the object being imaged. To enable such usage of the full screenwithout the use of a separate controller device, the user can entercommands into a user interface on the screen, and after a time delay(that can be set by the user), the mobile device can enter a measurementmode in which the patterns are displayed and the images are captured.Sounds and/or visual cues can also be used to alert the user when theimage capture commences and is completed such that the user does notprematurely move the mobile device while image capture is taking place.In another alternative embodiment, a separate server may not be used. Insuch an implementation, data processing can be performed on the mobiledevice 205 or the controller device 210 (if used).

In another alternative embodiment, two mobile devices may be used toform the proposed system. A first of the mobile devices is positionedwith its display facing the sample that is to be imaged, and is used toproject pattern(s) onto the sample. The second mobile device ispositioned with its rear-facing camera(s) toward the sample beingimaged. The rear-facing camera(s) are used to capture the images of thesample, based on the reflections of the patterns projected by the firstdevice. As a result, the images can be captured with better resolutionbecause rear-facing cameras typically have improved resolution ascompared to front-facing cameras in mobile devices. In such animplementation, the second device can also be used as the controller.Additionally, a special mount can be used to hold the two devicesrelative to one another and the sample. The mount can include a firstreceptacle configured to receive and hold the first mobile device, and asecond receptacle configured to receive and hold the second mobiledevice. When mounted in the mount, the displays of the first and seconddevices are facing in opposite (or near opposite) directions.

The front-camera objectives of mobile device cameras commonly have ashort focal length, which results in a large field of view.Unfortunately, this large field of view cannot be exploited in itsentirety by the proposed system. A valid PMD measurement can only betaken at image pixels that observe a display pixel over the specularsurface. This is because the mobile device cannot be held closer to theobject surface than the minimum possible focus distance and the screen(e.g., liquid crystal display (LCD) screen) has a limited angularcoverage. As a result, the number of pixels that produce validmeasurements can be as small as 25% of the imaging field of view. FIG.4A illustrates this problem with a planar object (plexiglass plate),which is placed at the minimum possible distance to the mobile device.Specifically, FIG. 4A depicts horizontal fringes within the effectivemeasurement field for a planar object (i.e., plexiglass plate) inaccordance with an illustrative embodiment. The plexiglass plate islarger than the field of view of the camera. Nevertheless, the patterncan only be observed in a small portion of the field of view. FIG. 4Bdepicts a horizontal fringe pattern reflected from the surface ofplexiglass plate in accordance with an illustrative embodiment. FIG. 4Cdepicts a calculated phase map based on the reflected horizontal fringepattern in accordance with an illustrative embodiment. FIG. 4D depicts acalculated gradient map after high pass filtering in accordance with anillustrative embodiment.

The inventors have also used the proposed system to evaluate the surfacenormal map of stained glass test tiles. FIG. 5A depicts a first stainedglass test tile in accordance with an illustrative embodiment. FIG. 5Bdepicts a second stained glass test tile in accordance with anillustrative embodiment. FIG. 5C depicts a third stained glass test tilein accordance with an illustrative embodiment. FIG. 5D depicts a fourthstained glass test tile in accordance with an illustrative embodiment.The stained glass test tiles have an approximately square shape withedge length of about 50 mm. The surface structure complexity and angulardistribution of surface normals increase from the sample of FIG. 5A tothe sample of FIG. 5D.

Most of the tiles in the test set display a size and surface normaldistribution small enough to be evaluated from a single view. FIG. 6Adepicts a surface normal map calculated from measurements of the firststained glass test tile in accordance with an illustrative embodiment.FIG. 6B depicts a surface normal map calculated from measurements of thesecond stained glass test tile in accordance with an illustrativeembodiment. FIG. 6C depicts a surface normal map calculated frommeasurements of the third stained glass test tile in accordance with anillustrative embodiment. FIG. 6D depicts a surface normal map calculatedfrom measurements of the fourth stained glass test tile in accordancewith an illustrative embodiment. For illustration purposes, themeasurement process is described for the ‘easiest’ test tile (i.e., FIG.5A). The test tile is placed at a position in the field of view of thefront-facing camera of the mobile device, where the reflected display ofthe mobile device can be observed. The intensity in each image pixel(x′,y′) can be expressed as:

I(x′,y′)=A(x′,y′)+B(x′,y′)·cos(ϕ(x′,y′)).  Eq. 1:

As shown above, equation 1 contains three unknowns per pixel. The(desired) phase ϕ(x′,y′) of the sinusoidal pattern, which correlatesdisplay pixels with image pixels is a first unknown. Additional unknownsare A(x′,y′) and B(x′,y′), which contain information about the unknownbias illumination and object reflectivity. This means that at leastthree equations are required to calculate ϕ(x′,y′). For each patterndirection, these equations are taken from the four acquired phase-shiftimages, where the intensity in each image pixel for the m^(th) phaseshift is:

I _(m)(x′,y′)=A(x′,y′)+B(x′,y′)·cos(ϕ(x′,y′)−ϕ_(m)), where:  Eq. 2:

ϕ_(m)=(m−1)π/2.  Eq. 3:

Finally, ϕ(x′,y′) can be evaluated by:

ϕ(x′,y′)=arctan(I ₂(x′,y′)−I ₄(x′,y′))/(I ₁(x′,y′)−I ₃(x′,y′)).  Eq. 4:

This procedure is performed for each pattern direction, leading to phasemaps ϕ_(x) (x′,y′) and ϕ_(y) (x′,y′) for the horizontal and verticalfringe direction, respectively. A phase map for the horizontal fringedirection is shown in FIG. 4C (image cropped for better visualization).This phase map is equivalent to the surface gradient in the verticaldirection plus a low frequency phase offset that is dependent on therelative position between camera and object, and any distortion presentin the camera objective. In conventional PMD setups, this offset isremoved by employing a calibration process whereby the phase map isfirst measured for a planar mirror, then subtracted from the measuredphase. A calibration process could also be performed for the systemsdescribed herein. However, for the desired qualitative measurements ofobjects like stained glass artworks, one can avoid this operation andthe respective phase calibration by exploiting a-priori knowledge aboutthe objects being imaged. For the example of stained glass, it is knownthat the overall shape is always flat.

As a result of this a-priori knowledge, the unknown phase offset can beremoved by high pass filtering the unwrapped phase map. The high passfiltered phase maps {tilde over (ϕ)}_(x) and {tilde over (ϕ)}_(y) arethen equivalent to the surface gradient maps in x- and y-direction. Itis noted that the filtering operation also compensates for the nonlinearphotometric responses of the display and camera, avoiding an additionalcalibration procedure. Moreover, the assumption of a flat objectresolves the depth-normal ambiguity of deflectometry measurements, whichtypically requires 2 cameras to resolve. The resulting horizontalgradient map is displayed in FIG. 4D. The results were calculated usingthe pattern frequency ν=1, corresponding to one sinusoidal perioddisplayed over the entire width of the display screen. Measurements withfrequencies ν>1 and subsequent multi-frequency phase unwrapping can beperformed as well. These measurements are not described in detail hereinbecause the approach does not significantly improve the performance of3D surface measurements and involves more time to acquire the necessaryfringe-images.

To image objects which are not flat, it is apparent that the assumptionof flatness cannot be used to avoid manual calibration. Rather, ageneral guess at the surface shape is made, and measurements are made todetermine whether the measurements confirm that the guess was accurate.If the measurements confirm that the guess is accurate, self-calibrationcan be used (i.e., the assumed information regarding the shape is theguess). If the measurements indicate that the guess is inaccurate,subsequent guesses are made until the measurements confirm a correctguess.

The surface normal can be computed directly from the estimated phasemaps via Equation 5 below:

$\begin{matrix}{{\overset{\rightarrow}{n} = {\frac{1}{\sqrt{\varnothing_{x}^{2} + \varnothing_{y}^{2} + 1}} \cdot \begin{pmatrix}\varnothing_{x} \\\varnothing_{y} \\{- 1}\end{pmatrix}}},} & {{Eq}.\mspace{14mu} 5}\end{matrix}$

where {tilde over (Ø)}_(x) and {tilde over (Ø)}_(y) denote the gradientfor the horizontal and vertical direction, respectively. As discussedabove, FIGS. 6A-6D depict the calculated normal maps of all 4 testtiles. The 4 normal maps are shaded with a specular finish and slightlytilted for visualization purposes. It can be seen that thecharacteristic surface structures are well resolved. The black spots inthe normal maps are produced by surface points where the surface normalresulted in no measured signal (i.e., the camera was not able to see thedisplay).

To test the robustness of the qualitative measurement results againstdifferent environmental conditions, additionally acquired measurementswere added for two of the four tiles with ambient room lighting and withperforming a hand-held measurement without mounting the device. Theresults are shown in FIG. 7. Specifically, FIG. 7A depicts normal mapmeasurement results with room light for the second and third stainedglass test tiles in accordance with an illustrative embodiment. FIG. 7Bdepicts normal map measurement results for the second and third stainedglass test tiles from handheld measurements in accordance with anillustrative embodiment.

The measurement taken with ambient room lighting (FIG. 7A) shows nosignificant degradation in performance. This is understandable becausethe brightness of the room light was moderate and the signal-to-noiseratio (SNR) was not reduced significantly. Under these conditions, thefour-phase shift algorithm effectively compensates for biasillumination. For the free-hand guided measurement, motion artifacts inthe evaluated phase map are expected. These artifacts can be seen at theslightly blurred edges in FIG. 7B. The fact that the visible artifactsoccur only at edges is a consequence of the low frequency ν=1 used toacquire these measurements. Higher frequencies may result in moreprominent artifacts, as for example commonly observed intriangulation-based fringe projection.

A single view measurement is often not enough to capture an extendedspecular object with large normal variation in its entirety. This is notonly because of the limited effective field of view of mobile devices,but also because the large normal variation of some surfaces cannot becaptured from a single viewing angle. A solution to this problem is toacquire and register multiple views of the object being imaged.Described below are qualitative results that demonstrate the feasibilityof this approach. A circular shaped glass painting with a diameter of300 mm was imaged. FIG. 8 depicts the circular shaped stained glassstructure imaged using registration in accordance with an illustrativeembodiment. From the magnification window in FIG. 8, it can be seen thatthe glass pieces in this stained glass structure exhibit high frequencysurface features. Moreover, some glass pieces are milky, whichintroduces an additional challenge for the surface measurement. For theresults shown below, one half of the glass structure was imaged byacquiring 14 single views under different viewing angles and positions.

To assist in registration, an additional ‘white image’ (i.e., an imageof the glass illuminated only by diffuse room light) of the glass wasacquired at each viewing position. The registration transformation forthe normal maps acquired at each single view was calculated from thesewhite images. Performing registration with the white images was found tobe more robust than registration with calculated normal maps. Forregistration, feature based registration algorithms provided by theMatlab Computer Vision Toolbox were used. In alternativeimplementations, different registration algorithms may be used. It isnoted that the use of images which are captured under diffuseillumination is beneficial in this case, since the diffuse illuminationmakes the object look similar from different viewing angles.Additionally, no strong specular reflections (which look different fromdifferent viewing angles) disturb the feature extraction of theregistration algorithm. Using this technique, subsequent views were ableto be registered without applying markers or other fiducials onto theobject surface (i.e., registration was performed just by using thetexture of the object itself).

FIGS. 9A-9D depict the registration results. FIG. 9A shows a capturedwhite image from a first view in accordance with an illustrativeembodiment. FIG. 9B shows a captured white image from a second view inaccordance with an illustrative embodiment. FIG. 9C depicts detected andmapped features in the two white images in accordance with anillustrative embodiment. FIG. 9D depicts the registered white images inaccordance with an illustrative embodiment.

It can be seen that the feature extraction and the subsequentregistration transformation is applied on the whole field of view of thecamera (not only on the limited field in the middle) in order to detecta large number of features with high quality. In this case it may bebeneficial to perform a calibration of the front camera (e.g. with acheckerboard) to compensate for distortion. This reduces theregistration error significantly. It should be also noted that such adistortion correction was avoided for the previous single-viewmeasurements, since most of the signal was measured in the middle of thefield of view, where the distortions are small.

The measurement results from the registration procedure described aboveare displayed in FIG. 10. FIG. 10A depicts a normal map of the stainedglass surface acquired from a first viewpoint in accordance with anillustrative embodiment. FIG. 10B depicts a normal map of the stainedglass surface acquired from a second viewpoint in accordance with anillustrative embodiment. FIG. 10C depicts a normal map of the stainedglass surface under a first alternative shading in accordance with anillustrative embodiment. FIG. 10D depicts a normal map of the stainedglass surface under a second alternative shading in accordance with anillustrative embodiment. As discussed above, the normal map is thecompilation of 14 single views which are stitched together viaregistration.

It can be seen from the views of FIG. 10 that the 3D surface isrecovered beyond the region in the center of the field of view.Depending on the orientation of the glass surface relative to camera anddisplay, it is principally possible to detect normals within the wholefield of view, though they might be sparse. As shown in the alternativeshading views of FIGS. 10C and 10D, most parts of the object's surfaceare densely reconstructed and the high frequency structures of theindividual glass pieces are visible. However, some black regions arestill present, mostly from the blue glass pieces in the painting. Thestructure of these pieces displays extraordinary high hills and deepcraters, producing a wide distribution of normals that can be measuredeffectively using more than the 14 views of the present example.

As discussed above, the proposed methods and systems are not limited tothe 3D measurement of stained glass artworks. A 3D surface acquisitionusing the proposed uncalibrated method is possible with virtually anysurface for which the overall shape of the object is flat and thesurface under test is relatively shiny. FIG. 11 depicts the surfacemeasurement of an oil painting. FIG. 11A is an image of the paintingbeing imaged in accordance with an illustrative embodiment. The surfacenormals of the black region in the dashed box (approximately 70 mm×80mm) was acquired using the proposed system. FIG. 11B depicts a firstview of the surface shape of the marked region in FIG. 11A in accordancewith an illustrative embodiment. FIG. 11C depicts a second view of thesurface shape of the marked region in FIG. 11A in accordance with anillustrative embodiment. In FIGS. 11B and 11C, the z-component wasexaggerated for display purposes. The surface shapes were calculated byintegration of the acquired normal map. Specifically, for a bettervisualization of the hills and valleys of the brushstrokes, the acquirednormal map was integrated to a depth map, using the Frankot-Chellappasurface integration algorithm. In alternative implementations, adifferent integration algorithm may be used. It can be seen that thebrushstrokes and canvas are nicely resolved in the surface shape imagesusing the proposed system.

It should be noted that the three-dimensional analysis of paintingsurfaces is of great interest for the cultural heritage community. Theability to separate surface texture from its shape or slope data is animportant tool for the analysis of painting techniques (e.g. by lookingat the directions of brush strokes) and monitoring of pigmentdegradation in paintings. The proposed mobile imaging methods andsystems are well suited for the analysis of paintings in the wild (e.g.,directly on a museum wall).

Another potential field of application of the proposed methods andsystems is the 3D acquisition of technical metallic surfaces.Measurement examples of such surfaces are shown in FIG. 12. FIG. 12Adepicts an image of a key in accordance with an illustrative embodiment.FIG. 12B depicts a measured normal map of the key of FIG. 12A inaccordance with an illustrative embodiment. FIG. 12C depicts normal mapsof a nickel and a dime in accordance with an illustrative embodiment.FIG. 12D depicts an image of a circuit board in accordance with anillustrative embodiment. FIG. 12E depicts a measured normal map of thecircuit board of FIG. 12D in accordance with an illustrative embodiment.As shown, imprinted letters and symbols are resolved, both for the keyas well as for the coins, using the proposed system. In the image of thecircuit board, the diameter of one single metallic ring is only about 2mm, and it can be seen that these rings are well resolved.

Another demonstrated the capability of the proposed system to measurefluid surfaces (e.g., for the analysis of surface tension). FIG. 13Adepicts a pattern of water drops on an enameled surface in accordancewith an illustrative embodiment. FIG. 13B depicts a normal map of thepattern of water drops depicted in FIG. 13A in accordance with anillustrative embodiment. As shown, the shape of each drop and theoverall pattern of drops are both clearly visible from the normal map.

Many of the imaging applications described herein are made feasible ormore convenient by incorporation of a mobile device into the system. Forexample, as discussed above, it can be difficult or impossible toconduct a 3D surface analysis of a rare painting without the use of amobile device. However, in alternative embodiments, any of theoperations and techniques described herein can be used with a stationarycomputing device (e.g., desktop computer, kiosk computer, permanentlymounted computing device, etc.) that includes a screen and a camera, asdescribed herein.

It is to be understood that any of the operations/processes describedherein may be performed at least in part by a computing system thatincludes a processor, memory, transceiver, user interface, etc. Thedescribed operations/processes can be implemented as computer-readableinstructions stored on a computer-readable medium such as the computersystem memory. Upon execution by the processor, the computer-readableinstructions cause the computing system to perform theoperations/processes described herein.

The word “illustrative” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“illustrative” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Further, for the purposes ofthis disclosure and unless otherwise specified, “a” or “an” means “oneor more.”

The foregoing description of illustrative embodiments of the inventionhas been presented for purposes of illustration and of description. Itis not intended to be exhaustive or to limit the invention to theprecise form disclosed, and modifications and variations are possible inlight of the above teachings or may be acquired from practice of theinvention. The embodiments were chosen and described in order to explainthe principles of the invention and as practical applications of theinvention to enable one skilled in the art to utilize the invention invarious embodiments and with various modifications as suited to theparticular use contemplated. It is intended that the scope of theinvention be defined by the claims appended hereto and theirequivalents.

What is claimed is:
 1. A three-dimensional (3D) imaging systemcomprising: a mobile device that includes: a display screen configuredto display a series of patterns onto an object that is to be imaged; anda front-facing camera configured to capture reflections of the series ofpatterns off of the object; a controller that is configured to control atiming of the series of patterns that appear on the display screen andactivation of the front-facing camera in relation to the appearance ofthe series of patterns.
 2. The system of claim 1, further comprising aserver that is in communication with one or more of the mobile deviceand the controller, wherein the server is configured to process thecaptured reflections to generate a normal map of at least a portion ofthe object.
 3. The system of claim 2, wherein the server is configuredto integrate the normal map to determine a surface shape of at least theportion of the object.
 4. The system of claim 3, wherein the serverintegrates the normal map into a depth map using a surface integrationalgorithm.
 5. The system of claim 3, wherein the server is configured toanalyze the surface shape of the object to determine a source of theobject.
 6. The system of claim 5, wherein the object comprises a stainedglass surface or a painting.
 7. The system of claim 2, wherein theserver is configured to process a plurality of sets of capturedreflections to generate a plurality of normal maps, wherein each set ofcaptured reflections corresponds to a different position of the mobiledevice relative to the object.
 8. The system of claim 7, wherein theserver is configured to combine the plurality of normal maps via aregistration procedure to generate a combined normal map that representsan entire surface of the object.
 9. The system of claim 8, wherein theserver performs the registration procedure using a plurality of whiteimages, wherein each white image corresponds to the different positionof the mobile device relative to the object.
 10. The system of claim 9,wherein the plurality of white images are captured under diffuseillumination of the object.
 11. The system of claim 1, furthercomprising a mount configured to maintain the mobile device at aspecific orientation relative to the object.
 12. The system of claim 1,wherein the series of patterns comprises three or more phase-shiftedsinusoidal patterns.
 13. The system of claim 1, wherein the series ofpatterns comprises a series of dot patterns, a series of line patterns,or an intensity gradient.
 14. The system of claim 1, further comprisinga processor configured to self-calibrate the system based on anassumption regarding a shape of the object.
 15. The system of claim 14,wherein the assumption regarding the shape of the object is that theobject is flat.
 16. A method for performing three-dimensional (3D)imaging, the method comprising: displaying, on a display screen of amobile device, a series of patterns onto an object that is to be imaged;capturing, by a front-facing camera of the mobile device, reflections ofthe series of patterns off of the object; controlling, by a controllerthat is in communication with the mobile device, a timing of the seriesof patterns that appear on the display screen and activation of thefront-facing camera in relation to the appearance of the series ofpatterns.
 17. The method of claim 16, further comprising processing, bya server in communication with one or more of the mobile device and thecontroller, the captured reflections to generate a normal map of atleast a portion of the object.
 18. The method of claim 17, furthercomprising: processing, by the server, a plurality of sets of capturedreflections to generate a plurality of normal maps, wherein each set ofcaptured reflections corresponds to a different position of the mobiledevice relative to the object; and receiving, by the server, a pluralityof white images, wherein each white image corresponds to the differentposition of the mobile device relative to the object; and combining, bythe server, the plurality of normal maps via a registration procedure togenerate a combined normal map that represents an entire surface of theobject, wherein the server performs the registration procedure using theplurality of white images.
 19. A three-dimensional (3D) imaging systemcomprising: a first mobile device that includes a display screenconfigured to display a series of patterns onto an object that is to beimaged; and a second mobile device that includes: a rear-facing cameraconfigured to capture reflections of the series of patterns off of theobject; and a controller that is configured to control a timing of theseries of patterns that appear on the display screen and activation ofthe rear-facing camera in relation to the appearance of the series ofpatterns.
 20. The system of claim 19, further comprising a mountconfigured to hold the first mobile device and the second mobile devicesuch that the display screen of first mobile device faces an oppositedirection of a display screen of the second mobile device.